Using a Chunk-based Dependency Parser to Mine Compound Words from Tweets

New words are appearing everyday in online communication applications, such as Twitter1. Twitter is the world’s most famous online social networking and microblogging service that enables its users to send/read text-based messages of up to 140 characters, known as “tweets”. Due to the facts that tweets are online typed (as fast as possible) within a limited number of characters, tweets are full of hand-made abbreviations and informal words. These facts make a difference between tweets and frequently used texts in regular web pages, such as news, blogs. Consequently, traditional hand-made corpora (in domains such as news) for natural language processing, such as word segmentation, partof-speech (POS) tagging, parsing, need to be “domain adapted” to be well suitable to tweets. That is, if one Japanese new (compound) word is not successfully recognized by a word segmentation toolkit, we can hardly ensure the word been well covered by a Japanese Input Method Editor (IME) or well translated by a statistical machine translation system. In this paper, we focus on novel compound word detection from Japanese tweets. we propose a method for mining contiguous compound words from single/double Bensetsus generated by a state-of-the-art chunk-based dependency parser, Cabocha2 (Kudo and Matsumoto, 2002) which makes use of Mecab3 with IPA dictionary4 for Japanese word segmentation, POS tagging, and